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The  evolution  of  test  and  evaluation  (T&E)  policy,  begun 
by  the  1970  Blue  Ribbon  Defense  Panel  (BRDP)  report,  is 
impacting  on  all  systems  acquisition  programs.  It  is 
imperative  that  today's  program  managers  no  longer  view 
testing  as  an  area  to  trade  off  when  the  dollar  and  time 
crunch  come  along.  Awareness  of  congressional  interest 
(PL-92-156),  Office  of  the  Secretary  of  Defense  direction 
(DODD  5000.1  and  5000.3)  and  Air  Force  guidance  (AF  Reg 
23-26  and  AF  Reg  80-14)  is  mandatory  in  the  development  of 
each  new  weapons  system.  It  is  the  consensus  of  informed 
people  in  the  T&E  community  that  today's  policies  will  be 
relatively  stable  over  the  next  few  years. 

I recommend  that  each  program  office i 

1.  Track  the  current  laws,  directives  and  regulations 
concerning  test  and  evaluation. 

2.  Provide  detailed  planning  for  all  testing  phases 
early  in  its  program  life  cycle. 

3.  Include  all  concerned  commands  and  agencies  in  all 
the  planning  phases. 

4.  Construct  a test  plan  which  will  provide  convincing 
proof  that  its  system  will  do  the  mission  it  is  designed  to 
do. 

5.  Assure  that  sufficient  numbers  of  its  system  are 
designed  into  its  full  scale  development  program  to  keep 


the  contractors’  production  and  engineering  personnel  in- 
tact until  IOT&E  is  complete  and  the  production  decision  is 
made. 

. In  today's  environment  you  must  sell  each  program 
through  adequate  and  convincing  testing. 
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THE  TEST  AND  EVALUATION  EVOLUTION 
(RELATIVE  TO  THE  U S AIR  FORCE) 


Introduction 


About  7 i OOO  people  showed  up  on  the  afternoon 
of  July  30,  (1909)  which  was  less  than  usual.  Rain 
showers  discouraged  many  from  making  the  trip  because 
they  knew  that  the  Wrights  never  flew  if  the  weather 
was  unfavorable.  About  four  o'clock  it  looked  as 
though  the  sky  was  clearing  and  the  wind  was  dying 
down.  Orville  told  the  board  members  he  would  be 
ready  in  about  one  hour  and  a half.  Maj.  Charles  McK. 
Saltzman  and  Lieut.  George  C.  Sweet,  a Navy  observer, 
quickly  drove  to  Shooter's  Hill  with  the  field  tele- 
phone to  let  us  know  when  everything  was  ready  there. 
The  Wrights  pushed  their  machine  to  the  starting  rail 
and  made  many  adjustments  to  the  engine  and  guy  wires. 
I put  two  stop  watches  around  my  neck  and  got  into 
the  passenger  seat.  I strapped  a box  compass  to  my 
left  thigh,  lashed  an  aneroid  barometer  to  my  right 
thigh,  and  jammed  a map  into  my  belt. 


Orville  warmed  up  the  engine  until  he  was 
satisfied  with  it,  and  climbed  aboard.  "If  I have  any 
trouble",  he  shouted  above  the  roar  of  the  engine, 

"I'll  land  in  a field  or  the  thickest  clump  of  trees 
I can  find." 

I nodded  and  gulped.  I had  picked  a course  with 
no  fields  of  any  kind  en  route.  It  was  too  late  to 
do  anything  about  it  now,  so  I grabbed  the  edge  of 
the  seat  with  both  hands  and  waited.  Orville  revved 
up  the  engine,  released  the  trigger,  and  the  machine 
started  down  the  rail.  We  skimmed  over  the  grass  for 
a few  feet  to  gain  speed,  and  then  climbed  for  altitude. 
As  we  started  to  circle,  Wilbur  ran  to  the  center  of 
the  field  below  us  with  a stop  watch  in  one  hand  and 
a signal  flag  in  the  other.  We  made  two  complete 
circles  of  the  field,  gaining  altitude  (125  feet), 
and  then  Orville  swung  sharply  over  the  starting  line. 
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I flicked  one  stop  watch  and  pointed  out  the  exact 
course  we  should  follow  to  Shooter's  Hill. 

All  twenty-five  horses  in  the  engine  were  func- 
tioning perfectly  as  we  skimmed  over  the  treetops 
toward  the  first  balloon.  The  air  was  bumpy,  and  I 
had  the  feeling  that  there  were  moments  when  Orville 
didn't  have  full  control  of  the  machine  as  we  dipped 
groundward.  It  was  as  if  someone  on  the  ground  had 
a string  attached  to  us  and  would  pull  it  occasionally 
as  they  would  a kite.  But  each  time  Orville  would 
raise  the  elevators  slightly,  and  we  would  gain  back 
the  lost  altitude. 

i 

We  reached  Shooter's  Hill  all  right,  and  I 
flicked  the  second  stop  watch.  There  was  a crowd  on 
the  brow  of  the  hill,  and  I could  see  them  wave  their 
umbrellas  and  handkerchiefs.  It  seemed  to  me  that 
the  angle  of  bank  of  the  plane  was  awfully  steep  as 
we  rounded  the  turn  and  the  wing  tip  was  much  too 
close  to  the  tops  of  the  trees.  A down  draft  hit  us, 
and  I thought  we  were  going  to  cartwheel  into  them 
for  sure.  We  straightened  out,  however,  and  started 
back  for  Myer.  Going  down  wind  now,  our  ground 
speed  increased  and  Orville  climbed  until  we  reached 
400  feet  - a world's  altitude  record.  As  we  neared 
Myer,  Orville  nosed  down  to  pick  up  speed,  and  aimed 
at  the  starting  tower.  I flicked  the  stop  watch 
off  as  we  crossed  the  starting  line  and  relaxed  as 
he  made  a circle  over  Arlington  Cemetery,  cut  off 
the  engine,  and  glided  in  for  a fairly  smooth  land- 
ing amid  a cloud  of  dust. 

Wilbur  rushed  up  to  us,  and  it  was  the  first 
time  I ever  saw  him  with  a smile  on  his  face.  I 
learned  later  that  he  had  experienced  some  excrucia- 
ting moments  of  doubt  when  we  had  disappeared  below 
the  level  of  the  trees  around  the  parade  ground  on 
our  outward  trip. 

The  crowd  was  larger  when  we  landed  than  when 
we  had  departed.  President  Taft  had  not  been  present 
for  the  takeoff  but  had  seen  us  land,  and  sent  a 
messenger  through  the  crowds  to  us  with  a note  of 
congratulation.  I had  the  feeling,  though,  that  some 
of  the  citizens  present  were  disappointed.  From  my 
experience  with  the  crowds  that  had  witnessed  the 
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previous  test  flights,  I had  no  doubt  that  many  of 
them  were  disappointed  that  we  had  not  landed  in 
Arlington  Cemetery  and  thus  provided  them  with  a real 
old-fashioned  Roman  holiday  with  all  the  bloody 
trimmings. 

As  soon  as  the  board  members  could  get  together, 
we  compared  our  stop  watches  and  determined  that  the 
official  speed  to  Alexandria  had  been  37.735  miles 

5er  hour;  on  the  return  trip  it  was  calculated  at 
7,431  miles  per  hour,  with  the  average  officially 
computed  at  42.583  miles  per  hour.  Major  Squier 
asked  the  Wrights  if  they  wanted  to  make  another 
trial  since  the  specification  allowed  them  three 
chances.  They  replied  that  they  would  stand  on  this, 
their  first  cross-country  flight. 

Since  they  had  flown  more  than  42^  miles  per 
hour,  it  was  thought  that  they  would  be  eligible  to 
collect  a bonus  of  $7,500;  however,  some  government 
lawyer  disallowed  the  fraction  above  the  42  miles 
per  hour  and  they  qualified  for  only  $5,000  above 
their  bid  price  of  $25,000.  On  August  2,  1909, 
Aeroplane  No.  1 was  officially  accepted  into  the 
inventory  of  the  United  States  Army.  ( 35*63-65) 


With  this  flight,  as  aptly  described  by  Benjamin  Foulois, 
the  first  aircraft  test  and  evaluation  program  was  success- 
fully completed.  A new  era  was  begun  - military  aviation. 
This  also  marked  the  beginning  of  military  test  and  evalua- 
tion of  aircraft.  That  policies  and  procedures  have 
changed  is  obvious  to  anyone  even  slightly  aware  of  today's 
methods  of  systems  acquisition.  What  may  not  be  as  obvious 
are  the  changes  that  have  occurred  in  just  the  last  four 
years  and  that  continue  to  occur.  My  purpose  here  is  to 
trace  the  recent  evolution  of  test  and  evaluation,  where 
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appropriate,  limiting  the  scope  to  the  U S Air  Force,  and 
further  to  aircraft  testing. 

In  the  next  section  of  this  paper  I will  review  recent 
guidance  and  directives  to  bring  you  up  to  date. 

Following  this  analysis  of  the  current  state,  the  next 
two  sections  will  address  the  direction  of  test  and 
evaluation  in  the  future.  First  by  revealing  the  results 
of  numerous  interviews  and  then  by  analyzing  and  inter- 
preting this  information. 

Before  continuing,  I want  to  draw  your  attention  to 
Appendix  A where  you  will  find  a glossary  of  terms  defined 
as  they  are  used  in  the  test  business.  Unless  you  are  on 
speaking  terms  with  the  current  T & E terminology,  you 
will  find  this  a few  minutes  well  spent. 
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REVIEW  OF  LITERATURE 


Aircraft  testing  has  been  with  us  since  the  acceptance 
of  Aeroplane  No.  1,  as  described  in  the  introduction,  but 
the  how,  what  and  who  of  this  testing  have  changed  as 
drastically  as  the  aircraft's  speed  and  altitude  have.  In 
today's  environment  of  fewer  dollars  and  the  high  rate  of 
change  of  our  technical  capability,  testing  has  gained  a 
new  role  of  importance.  To  understand  both  the  reasons  for 
and  the  implications  of  this  role  I find  it  most  helpful  to 
review,  in  brief,  the  documents,  reports,  directives, 
regulations  and  policy  guidance  in  the  order  in  which  they 
appeared.  Table  I is  a chronological  listing  of  the  major 
documents  beginning  with  the  Blue  Ribbon  Defense  Panel  (BRDP) 
report  in  July  of  1970  up  through  the  draft  revision  to 
Air  Force  Regulation  80-14,  "Test  and  Evaluation". 

Although  there  is  some  record  of  concern  over  test  and 
evaluation  (T&E)  policy  and  procedure  prior  to  the  BRDP 
report  it  undoubtedly  provided  the  major  thrust  behind  the 
changes  that  have  taken  place  over  the  last  four  years  and 
that  are  continuing  today.  This  group  of  high  level  men 
took  an  in-depth  look  at  all  phases  of  the  operation  of  the 
Department  of  Defense  at  the  request  of  President  Nixon.  A 
synopsis  of  their  findings  in  the  area  of  Operational 
Testing  and  Evaluation  (OT&E)  is  as  follows: 
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Functional  testing  (often  called  engineering 
testing)  is  done  to  determine  how  well  various 
systems  and  materiel  ...  meet  technical  require- 
ments. 

By  and  large,  functional  testing  in  and  for  the 
Department  of  Defense  appears  to  be  well  understood 
and  faithfully  executed.  ...  Functional  testing 
is  not  considered  to  be  a major  problem  area. 

Operational  testing,  on  the  other  hand,  is 
done  to  determine  to  the  extent  possible  whether 
such  systems  and  materiel  can  meet  operational 
requirements. 

There  has  been  an  increasing  desire,  particu- 
larly at  OSD  level,  to  use  data  from  OT&E  to  assist 
in  the  decision-making  process. 

Unfortunately,  it  has  been  almost  impossible 
to  obtain  test  results  which  are  directly  applicable 
to  decisions  or  useful ‘for  analyses.  Often  test 
data  do  not  exist.  When  they  do,  they  frequently 
are  derived  from  tests  which  were  poorly  designed 
or  conducted  under  insufficiently  controlled 
conditions  to  permit  valid  comparisons. 

The  most  glaring  deficiency  of  OT&E  is  the 
lack  of  any  higher-than-Service  organization 
responsible  for  overseeing  Defense  OT&E  as  a whole. 

There  are  three  principal  problems  with  Air 
Force  OT&E,  as  currently  done.  First,  operational 
considerations  receive  much  too  little  attention 
in  Categories  I and  II.  Second,  the  operational 
commands  responsible  for  Category  III  and  Opera- 
tional Employment  Testing  lack  both  the  personnel 
and  facilities  to  be  effective.  Finally,  all  of 
the  categories  are  tco  duplicative  and  time-consu- 
ming. 

Currently,  there  is  no  effective  method  for 
conducting  OT&E  which  cuts  across  Service  lines, 
although  in  most  actual  combat  environments,  the 
United  States  must  conduct  combined  operations. 
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Funding  throughout  the  Department  of  Defense 
has  been  and  continues  to  be  inadequate  to  support 
much  necessary  OT&E,  ...  In  fact,  there  is  no 
agency  that  can  even  identify  the  funds  that  are 
being  spent  on  OT&E,  (3*88  to  91) 

As  a result  of  these  findings  the  BRDP  made  the  follow- 
ing three  recommendations i 

The  responsibility  for  Defense  Test  and  evalua- 
tion policy  should  be  assigned  to  the  Assistant 
Secretary  of  Defense  (Test  and  Evaluation). 

A separate  program  category  should  be  estab- 
lished for  Test  and  Evaluation. 

The  responsibility  for  overview  of  Defense  test 
and  evaluation  effort  should  be  assigned  to  the 
Defense  Test  Agency.  In  addition,  the  Agency  should 
be  responsible  for  design  or  review  of  test  designs, 
performing  or  monitoring  of  tests,  and  continuous 
evaluation  of  the  entire  test  and  evaluation  program. 
(3*91) 

The  DOD  organization,  as  recommended,  is  shown  in  Chart 

I.  • (3«60) 

Now  the  ball  was  in  the  hands  of  the  DODj  if  they 
agreed  with  the  finding  of  the  BRDP  then  something  had  to 
change.  Well,  change  they  did,  under  the  continual 
guidance  and  prodding  of  the  then  Deputy  Secretary  of 
Defense,  David  Packard.  He  began  with  an  exchange  of  memos 
with  the  Service  Secretaries,  beginning  just  13  days  after 
the  BRDP  report  came  out.  After  hearing  from  each  of  the 
Service  Secretaries,  Mr.  Packard  directed  the  changes  he 
believed  necessary  to  satisfy  the  shortcomings  reported  by 
the  BRDP.  Then,  on  13  July  1971.  came  DOD  Directive  5000.1 
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with  change  in  many  areas,  establishing  the  Defense  System 
Acquisition  Review  Council  (DSARC)  and,  importantly,  re- 
quiring systems  demonstration,  to  the  extent  practical,  by 
actual  performance  testing  prior  to  moving  into  full  scale 
production. 

This  was  the  high  level  emphasis  that  OT&E  needed  and 

on  24  August  1971  the  Deputy  Director  for  Research  and 

Engineering  (DDR&E)  issued  a memo  establishing  a Test  and 

Evaluation  Office.  Lt.  General  Alfred  D.  Starbird,  USA  (RET) 

was  selected  as  its  head.  In  his  own  words  he  was  given 

across-the-board  responsibility  for  the  Office 
of  the  Secretary  of  Defense  in  test  and  evaluation 
matters.  I was i 

a.  To  keep  under  surveillance ' all  T&E  policy 
and  procedures  and  recommend  directly  to  the 
Secretary  of  Defense  any  changes  necessary. 

b.  To  monitor  closely  all  major  acquisition 
programs,  advising  the  DSARC  and  the  SecDef 
directly  at  key  decision  points  on  the  adequacy 
of  the  T&E  to  support  the  Service  action  proposed. 

c.  To  initiate  and  coordinate  accomplishment  of 
such  joint  testing  as  is  necessary. 

d.  To  oversee  for  OSD  its  major  ranges  and 
test  facilities,  ( 32 1 3 ) 

As  a result  of  criticism  of  service  testing  from  the 
Congress  and  most  likely  due  in  part  to  the  efforts  of  the 
late  Senator  Ellendar,  a bill  was  passed  directing  the 
Services  to  provide  the  results  of  OT&E  to  the  Congress, 
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Specifically  the  bill,  passed  November  17,  1971  states  in 
parti 

Sec. 306  (a)  Beginning  with  the  calendar  year 
1972,  the  Secretary  of  Defense  shall  submit  to  the 
Congress  each  calendar  year,  at  the  same  time  the 
President  submits  the  Budget  to  the  Congress  ... 
a written  report  ...  Beginning  with  the  calendar 
year  1973»  there  shall  be  included  in  the  report 
data  on  operational  testing  and  evaluation  for  each 
such  weapon  system  for  which  funds  for  procurement 
are  requested  (other  than  funds  requested  only  for 
the  procurement  of  units  for  operational  testing 
and  evaluation  and/or  long  lead-time  items), 

(b)  A supplemental  report  shall  be  submitted 

to  the  Congress  by  the  Secretary  of  Defense  not  less 
than  thirty  nor  more  than  sixty  days  before  the 
awarding  of  any  contract  or  the  exercising  of  any 
option  in  a contract  for  the  procurement  of  any  such 
weapon  system  (other  than  procurement  of  units  for 
operational  testing  and  evaluation  and/or  long  lead- 
time  items). 

(c)  Any  report  required. ., shall  include  detailed 
. and  summarized  information  with  respect  to  each 

weapon  system  covered  by  such  report,  and  shall 
specifically  include,  but  shall  not  be  limited  to 
...the  results  of  all  operational  testing  and 
evaluation  up  to  the  time  of  the  submission  of  the 
report,  or,  if  operational  testing  and  evaluation 
has  not  been  conducted,  a statement  of  the  reasons 
therefor  and  the  results  of  such  other  testing  and 
evaluation  as  has  been  conducted,  (2316  &7) 

The  Air  Force  published  a revision  to  its  major  test 

and  evaluation  regulation,  AFR  80-14,  on  12  May  1972.  The 

major  changes  were  to  reflect  the  new  policies  developed  by 

the  Office  of  the  Secretary  of  Defense  (OSD)  concerning 

the  conduct  of  test  and  evaluation  in  the  acquisition 

process.  Specifically,  it  changed  the  categories  of  testing 
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(Former  Cat  I and  Cat  II,  now  DT&E.  Former  Cat  III  now 
OT&E)  and  required  greater  participation  by  the  operating 
and  supporting  commands  in  test  and  evaluation. 

On  17  July  and  7 August  1972,  two  Comptroller  General 
reports  to  Congress  (GAO  reports)  were  published.  The 
first  was  concerned  with  the  acquisition  of  major  weapons 
systems;  the  second  with  the  importance  of  test  and  evalua- 
tion in  the  acquisition  process. 

Both  reports  substantiated  the  soundness  of  the 

evolving  DOD  T&E  policies.  The  first  by  suggestingi 

...that  DOD  policies  and  practices  regarding 
testing  consider! 

— Adequate  controls  over  granting  any  waivers 
from  required  testing  and  evaluation. 

— Completion  of  appropriate  testing  and  evaluation 
prior  to  key  decision  points  in  the  acquisition 
cycle.  (5*35) 

The  latter  by  recommending! 

...that  DOD,  in  implementing  its  new  policies 
and  practices  regarding  testing  and  evaluation, 
continue  to  emphasize  the  need  fori 

--Completion  of  appropriate  testing  and  evaluation 
prior  to  key  decision  points  in  the  acquisition 
cycle. 

--Adequate  control  over  granting  waivers  from 
required  testing  and  evaluation. 

— Succinct  summary  reports  to  be  prepared  by  the 
testing  agency  for  all  levels  of  management. 
Interested  management  levels  may  wish  to  comment 
on  these  summary  reports;  however,  they  should  not 
be  permitted  to  change  the  basic  summaries.  (6i4l) 
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Other  government  reports,  such  as  the  Report  of  the 
Commission  on  Government  Procurement  (Vol,2),  also  empha- 
sized the  importance  of  test  and  evaluation.  Everyone  was 
joining  the  chorus,  singing  the  praises  of  a properly 
written,  timely  and  independently  conducted  test  program. 

At  this  point,  however,  the  Air  Force  continued  to  tout 
their  test  centers  as  independent  test  agencies,  not 
responsible  to  the  System  Program  Offices,  though  a part  of 
Systems  Command,  This  was  to  change,  but  not  just  yet. 

In  January  of  1973  DOD  Directive  5000. 3*  "Test  and 
Evaluation"  was  published,  formalizing  and  replacing  the 
numerous  Deputy  Secretary  of  Defense  Memos  (11  Feb  1971 » 

21  April  1971  and  3 Aug  1971)  concerned  with  test  and  evalua 
tion.  This  continues  to  be  the  major  DOD  guidance  in  T&E. 

A brief  synopsis  of  5000.3*  as  outlined  by  General  Starbird 
in  his  18  July  1973  statement  to  Congress  provides  an  over- 
view of  the  major  points  of  this  directive. 

a.  It  stated  certain  principles  that  should 
apply  to  all  acquisition  programs.  T&E  would 
start  as  early  as  possible  ar.d  be  conducted  in 
phases  so  as  to  eliminate  early  the  risks. 
Acquisition  schedules  must  be  keyed  to  accom- 
plishing T&E  milestones  before  major  added 
resource  commitments  were  made.  Prior  to  con- 
tracting for  major  production,  there  would  be 
completed:  sufficient  development  testing  to 
insure  that  all  design  problems  were  identified 
and  solutions  in  hand;  and  sufficient  operational 
testing  to  provide  a valid  estimate  of  opera- 
tional effectiveness  and  suitability. 
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b.  OT&E  would  be  the  responsibility  of  a field 
agency  independent  of  the  developing  command. . . 
Such  operational  testing  was  to  be  conducted  in 

a simulated  operational  environment  where  some  of 
the  stresses  of  real  combat  operations  such  as 
fatigue  and  countermeasures  were  applied, 

c.  Operational  testing  preferably  was  to  be 
separate  from  development  testing.  However,  an 
early  operational  test  could  be  combined  with 

a development  test  providing!  separation  would 
cause  "delay  involving  unacceptable  military 
risks  or  unacceptable  increase  in  systems  cost"... 

d.  Pilot  production  items  were  to  be  used 
wherever  possible  for  the  development  and 
operational  testing  completed  just  prior  to 
commitment  to  major  production.  Again,  however, 
if  awaiting  such  pilot  production  would  involve 
"unacceptable  military  risk  or  unacceptable 
increase  in  total  acquisition  cost"  then  final 
prototypes  would  be  employed,  but  only  if  these 
prototypes  were  reasonably  like  the  expected 
production  items. 


f.  The  Directive  included  an  important  new 
provision  with  respect  to  waiving  satisfactory 
completion  of  required  T&E.  For  major  pro- 
grams the  DCP  sets  forth  such  required  T&E  and 
only  the  Secretary  of  Defense  can  waive  its 
accomplishment.  For  lesser  programs,  waiver  can 
be  granted  only  by  the  Secretary  of  the  Military 
Department  or  his  immediate  subordinate  designee. 

(32 1 4 & 5) 

The  provisions  of  DODD  5000,3  which  require  each  DOD 
Component  have  one  major  field  agency,  (or  a limited  number 
of  such  major  field  agencies)  separate  and  distinct  from 
the  developing/procuring  command,  responsible  for  OT&E 
created  a problem  for  the  Air  Force,  As  stated  earlier, 
they  considered  their  test  centers  independent.  Obviously 
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this  would  not  meet  the  requirements  of  DODD  5000.3.  In 
January  of  this  year  (1974)  Air  Force  issued  a Draft 
Regulation  (AFR  23-26)  establishing  and  implementing  the 
Air  Force  Test  and  Evaluation  Center  (AFTEC)  as  a separate 
operating  agency,  its  commander  reporting  directly  to  the 
Chief  of  Staff  Air  Force  (CSAF).  AFTEC's  organization  and 
mission  will  be  discussed  more  fully  in  a later  section  of 
this  paper. 

On  20  March  of  this  year  a Draft  Revision  AFR  80-14  (T&E) 
was  issued.  It  incorporated  changes  brought  about  by  the 
formation  of  AFTEC  and  details  the  responsibilities  and 
interfaces  of  AFTEC  with  implementing,  operating  and 
supporting  commands. 

The  foregoing  has  been  a brie-f  summary  of  the  evolution 
in  test  and  evaluation  over  the  past  four  years,  a most 
interesting  history  whose  impact  is  just  now  being  felt. 

Chart  2 shows  the  current  DOD  and  Air  Force  T&E  Organiza- 
tion. The  question  remains,  where  do  we  go  from  here? 
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CHRONOLOGICAL  ORDER  OF  TEST  & EVALUATION  DIRECTIVES 
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Secretaries.  (20) 

Joint  Memo  by  the  Service  Secretaries  to 
the  Secretary  of  Defense.  (16) 

Secretary  of  the  Air  Force  Memo  to  the 
Secretary  of  Defense.  (19) 

DDR&E  "Fly  Before  Buy  Procurement"  Memo.  (14) 

Secretarv  of  Defense  Memo,  "Conduct  of 
OT&E".  (21) 

Deputy  Secretary  of  Defense  Memo  to  the 
Secretaries  of  the  Military  Departments.  (15) 

AFSC  Pamphlet  800-3.  (D 

DODD  5000.1.  (11) 

Secretary  of  Defense  Memo  "Test  and  Evalua- 
tion in  Systems  Acquisition  Process".  (22) 

DDR&E  Memo  establishing  the  T&E  Office.  (17) 

PL-92-156  requiring  OT&E  reporting  to 
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Air  Force  Reg  80-14,  "Test  and  Evaluation" ( 9 ) 
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CHRONOLOGICAL  ORDER  OF  TEST  & EVALUATION  DIRECTIVES 


r 


TABLE  I CONTINUED 
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Comptroller  General  Report  to  Congress, 
"Importance  of  T&E"  . (6) 
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Comptroller  General  Report  to  the  House 
Armed  Services  Committee,  "Cost  Growth 
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"Test  and  Evaluation",  (10) 
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CHART 


INTERVIEWS 


* ^ 


Where  are  we,  the  DOD  and  specifically  the  Air  Force, 
going  from  here?  What  are  the  next  steps  in  this  test  and 
evaluation  evolution?  In  an  attempt  to  get  informed 
answers  to  these  questions;  answers  that  would  indicate 
both  the  areas  of  concurrence  and  those  yet  to  be  fully 
agreed  upon  by  the  major  levels  in  the  T&E  hierarchy,  I 
prepared  a short  questionnaire.  This  provided  both  a 
standard  for  comparison  and  a point  of  departure  for 
further  discussion.  The  questionnaire  used  is  reproduced 
on  the  next  page  and  is  followed  on  the  next  four  by  a 
brief  summary  of  the  answers  received.  Those  interviewed 
were: 

1.  Rear  Admiral  Forrest  S.  Petersen,  Assistant 
Director  (Strategic  & Support  Systems  Test  and 
Evaluation) , ODDR&E. 

2.  Lt  Colonel  James  Martin,  Hq.  Air  Force/AFRDPQ, 

3.  Colonel  Max  Findell,  HqAFSC/DOVM. 

4.  Major  Larry  Griffin,  AFTEC/TET. 

Expanded  answers  and  an  analysis  of  this  and  related 
information  will  be  found  in  the  next  section  of  this  paper. 
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QUESTIONNAIRE 


1.  Will  each  program  require  a testing  phase  prior  to  its 
production  go-ahead?  (If  not,  is  the  dollar  criteria 
the  same  as  for  DSARC?) 

2.  Is  this  testing  what  is  now  called  OT&E  or  IOT&E? 

3.  How  autonomous  should  the  Air  Force  OT&E  test  activity 
be?  (To  what  level  should  the  activity  report?) 

4.  What  organization  funds  the  independent  OT&E  agency? 

5.  What  role  will  the  newly-formed  AFTEC  play  in  aircraft 
OT&E? 

6.  Who  will  prepare  the  OT&E  test  plan  (for  major  aircraft 
systems)? 

7.  How  will  crew  members  be  selected  to  fly  the  OT&E 
' flight  test  plan? 

8.  Who  will  prepare  the  OT&E  test  report? 

9.  Where  does  it  go? 

10.  Are  there  any  major  systems  that  would  have  been  can- 
celled if  OT&E  had  been  completed  prior  to  the  beginning 
of  the  production  phase? 
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Answers  provided  by  Rear  Admiral  Peterson  (DDR&E) 


1.  Yes  - for  all  major  programs  and  at  least  in 
principle  for  those  less  than  major. 

2.  The  testing  prior  to  production  go-ahead  is  IOT&E. 

3.  To  the  Chief  of  Staff  Air  Force. 

4.  It  will  be  shared  by  AF3C , AFTEC  and  the  user. 

5.  Primarily  they  will  manage  the  test. 

6.  AFTEC 

7.  From  the  using  command  TDY  to  AFTEC. 

8.  AFTEC 

9.  Directly  to  the  Chief  of  Staff  Air  Force  with 
copies  to  all  interested  commands. 

10.  No 
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Answers  provided  by  Lt  Colonel  Martin  (Hq  Air  Force) 


1.  Each  program  will  have  a test  plan;  testing  can  be 
waived  per  DODD  500°. 3» 

2.  IOT&E,  testing  conducted  prior  to  the  production 
go-ahead.  DT&E  data,  when  reviewed  by  the  user, 
can  be  a portion,  or  even  all,  of  IOT&E. 

3.  Should  report  to  the  Chief  of  Staff  Air  Force. 

4.  This  is  an  item  in  the  Air  Force  budget. 

5.  AFTEC  will  manage  overall  IOT&E  as  well  as  OT&E 
programs . 

6.  AFTEC,  with  inputs  from  the  users. 

7.  Using  command,  to  criteria  set  by  AFTEC. 

8.  Inputs  given  by  the  test  team,  evaluated  by  AFTEC. 

9.  Directly  to  the  Chief  of  Staff  Air  Force  with 
copies  to  the  implementing,  operating  and 
supporting  commands.  DDR&E  (T&E)  has  access  to 
all  reports. 

10.  The  B-58. 
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Answers  provided  by  Colonel  Findell  (HqAFSC) 


1.  Yes 

2.  IOT&E 

3.  There  is  a dispersion  of  OT&E  responsibility 
between  AFTEC  and  the  operating  commands, 

AFTEC  reports  to  the  Chief  of  Staff  Air  Force 
directly. 

4.  IOT&E  primarily  by  AFSC , OT&E  by  the  operating 
commands . 

5.  AFTEC  may  manage  all  T&E  or  may  monitor  just  IOT&E. 

6.  The  test  centers  will  be  responsible  for  detailed 
test  planning. 

7.  They  will  come  from  the  operating  command. 

8.  Combined  AFTEC  and  operating  command  effort. 

9.  To  the  Chief  of  Staff  Air  Force  with  copies  to  all 
interested  parties, 

10.  Probably.  (None  named) 


Answers  provided  by  Major  Griffin  (AFTEC) 
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1.  Yes,  as  required  by  DODD  5000,3  and  AFR  80-14. 

2.  That  is  what  we  call  IOT&E. 

3.  A very  autonomous  operation,  reporting  directly 
to  the  Chief  of  Staff  Air  Force. 

4.  The  developing  and  using  commands j their  share 
depends  on  the  type  of  test. 

5.  It  will  be  as  required,  providing  the  test 
conductor,  test  manager  or  test  monitor, 
depending  in  part  on  AFTEC ' s capability. 

6.  Either  the  PMD  or  the  Test  Director  out  of 
Hq  Air  Force  will  deliniate  who  will  prepare 
the  test  plan. 

7.  Some  will  be  from  AFTEC,  primarily  on  component 
testing,  primarily  they  will  come  from  the 
operating  command. 

8.  The  Test  Director,  either  from  AFTEC  or  the 
command  doing  the  test.  All  will  go  through 
AFTEC  for  review, 

9.  To  AFTEC  for  review  and  a Commander's  Summary. 

It  will  then  go  directly  to  the  Chief  of  Staff 
Air  Force  with  no  changes  in  the  original  report. 

10.  The  C-5  would  have  been  changed,  but  not  cancelled. 
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ANALYSIS  OF  INTERVIEWS  AND  RELATED  INFORMATION 

The  interviews  reported  here  are  surprising  in  only  one 
respect  - they  were  in  almost  total  agreement.  Where  they 
differ  is  in  the  degree  of  involvement  of  AFTEC  in  test  and 
evaluation.  Looking  at  each  question  I find  a consensus 
as  follows » 

1.  Each  and  every  program  will  require  a testing 
phase  prior  to  its  production  go-ahead.  In  rare  cases  the 
test  requirements  can  be  waived,  by  the  Chief  of  Staff  Air 
Force  for  less  than  major  programs,  and  by  OSD  for  major 
programs , 

2.  The  current  name  for  the  testing  accomplished 
specifically  to  assist  in  making  the  DSARC  III,  production 
go-ahead  decision,  is  Initial  Operational  Test  and  Evalua- 
tion (IOT&E). 

3.  The  question  of  autonomy  was  answered  in  each  case 
by  pointing  out  that  the  test  agency  should  report  directly 
to  the  Chief  of  Staff  Air  Force. 

4.  At  this  point  in  time  the  understanding  is  that 
test  and  evaluation  funding  will  be  shared  by  the  devel- 
oping and  operating  commands,  with  AFSC  primarily  funding 
IOT&E  and  the  operating  command  primarily  funding  OT&E. 

5.  AFTEC  will  provide  a management  role  in  Air  Force 
T&E.  It  is  interesting  to  note  that  the  AFTEC  response  to 
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this  question  expanded  on  the  management  role  and  included 
the  probability  of  active  flying  participation. 

6.  AFTEC  will  have  final  approval  on  the  adequacy  of 
test  plans;  however,  the  actual  preparation  will  likely  be 
by  combining  inputs  from  all  concerned  parties, 

7.  Crew  members  for  aircraft  IOT&E  and  OT&E  flight 
tests  will  come  from  the  operating  command  with  the 
possibility  of  some  from  AFTEC. 

8.  All  test  reports  will  be  prepared  by  the  test 
director,  usually  from  AFTEC;  however,  he  could  be  from 
the  operating  command.  In  every  instance  the  report  will 
be  reviewed  by  AFTEC  and  a Commander's  Summary  will  be 
attached.  The  original  report  will  not  be  changed. 

9.  All  reports  will  be  sent  directly  to  the  Chief  of 
Staff  Air  Force  with  copies  to  the  implementing,  operating 
and  supporting  commands. 

10.  Opinions  varied;  however,  all  agreed  that  while 
program  cancellation  may  not  have  been  the  result  of 
current  T&E  policies  being  applied  to  past  programs, 
significant  changes  to  programs  such  as  the  C-5  and  B-58 
would  have  been  required. 

Colonel  James  Abrahamson,  former  Naverick  SPO  Director 
(currently  the  Systems  Command  Inspector  General),  has  a 
unique  insight  to  the  evolving  test  and  evaluation  policies. 
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During  his  tenure  as  Maverick  Director,  that  program  was 
selected  by  DDR&E  as  the  first  program  to  have  its  full 
production  go-ahead  delayed  pending  an  in-depth  opera- 
tional test  program.  DDR&E  wanted  an  independent  evalua- 
tion, the  goal  of  the  emerging  guidance,  and  at  this  time 
the  Air  Force  did  not  have  what  they  consider  to  be  an 
independent  test  agency.  They  went  to  other  government 
agencies  such  as  Weapons  Systems  Evaluation  Group  (WSEG) 
and  Institute  for  Defense  Analysis  (IDA)  for  test  direction. 
The  Air  Force  gave  money  to  DDR&E  to  buy  test  equipment 
and  instrumentation.  The  Maverick  Program  Office  provided 
both  the  modified  delivery  aircraft  and  Maverick  missiles 
to  be  tested.  The  test  plan  would  not  allow  AFSC  pilots 
to  .fly  the  test  program,  so  additional  time  and  funds  were 
required  to  train  TAC  pilots  to  fly  the  test  plan.  Major 
emphasis  was  placed  on  a realistic  test  environment  and 
joint  testing.  The  test  results  were  further  evaluated 
by  analyzing  the  results  in  simulation  against  a European 
weather  environment. 

Colonel  Abrahamson  views  this  as  an  example  of  how  AFTEC 
will  operate,  drawing  resources  from  the  implementing  and 
operating  commands  and  providing  overall  test  management 
and  an  independent  evaluation  directly  to  the  Chief  of 
Staff  Air  Force  and  OSD  at  their  request. 
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It  is  important  to  note  at  this  point  that  there  is  great 


concern  within  the  implementing  command  that  the  current 
test  and  evaluation  policies  will  cause  major  program  cost 
increases  by  enforcing  a delay  between  full  scale  devel- 
opment and  production.  This  concern  has  been  the  topic 
of  a major  study  effort  undertaken  by  the  Aeronautical 
Systems  Division  of  AFSC  under  the  direction  of  Lt  General 
James  Stewart. 

The  central  question  is  the  balance  between  what 
adequate  testing  saves  in  future  program  costs  and  the 
increased  costs  incurred  by  causing  a contractor  to  slow 
down  or  stop  his  manufacturing  effort  while  IOT&E  is 
conducted.  .This  subject  is  treated  in' an  individual  study 
paper  written  by  Lt  Colonel  G.  Kline,  titled  "The  Penalty 
of  an  Extreme  Incremental  Acquisition  Strategy".  Further 
information  in  this  area  can  also  be  found  by  referring 
to  the  ASD  study  mentioned  above.  (24) 

The  role  of  AFTEC  is  best  described  by  its  mission 
statement  as  follows: 

AFTECs  mission  is  to  manage  the  Air  Force's 
Operational  Test  and  Evaluation  (OT&E)  program. 

AFTEC  assesses  the  military  utility  of  major  and 
HQ  USAF-des imated  non-major  Air  Force  systems. 

Major  commands  ( MAJC0M.5 ) suoport  AFTEC  as  directed 
by  HQ  USAF  program  management  directives  (PMDs)  or 
test  directives  (TDs).  AFTEC  recommends  Air  Force 
OT&E  policy  to  HQ  USAF  for  aporoval  as  well  as 
plans,  directs,  controls,  evaluates,  and  reports-- 
independently--on  OT&E.  AFTEC  serves  as  the 
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principal  field  command  for  providing  OT&E  infor- 
mation to  the  Secretary  of  the  Air  Force  (SAF)  and 
the  Chief  of  Staff  of  the  Air  Force  (CSAF)  in 
preparation  for  Defense  Systems  Acquisition  Review 
Council  (DSARC)  actions  and  to  support  those  Air 
Force  procurement  requests  for  which  OT&E  infor- 
mation is  statutorily  required  to  be  supplied  to 
the  Congress.  (8il) 

As  to  the  future  a recent  high  level  speaker  at  the 
Defense  Systems  Management  School,  when  questioned  about 
further  implementation  of  the  BRDP  recommendations  in  the 
test  and  evaluation  area,  stated  that  OSD  would  probably 
support  a gradual  expansion  of  General  Starbird's  test 
and  evaluation  role  in  systems  acquisition;  however,  to 
support  a major  change  is  doubtful  and  would  probably  not 
be  supported.  This  is  also  a general  consensus  of  those 
I have  talked  with  regarding  the  future  direction  of  test 
and  evaluation  policy.  The  current  policies,  when  fully 
implemented  will  be  stable  for  some  years  to  come.  They 
will  continue  to  provide  the  confidence  we  seek  in  a new 
weapon  system  prior  to  a full  production  commitment. 
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CONCLUSIONS 


The  single  overriding  message  to  come  out  of  the  test 
and  evaluation  evolution  is  that  any  new  acquisition  program 
must  be  capable  of  meeting  or  exceeding  the  real  world  goals 
expected  of  it  in  the  actual  or  closely  simulated  opera- 
tional environment  prior  to  any  major  production  commitment. 
The  evolution  which  began  just  four  years  ago  should  convince 
even  the  most  skeptical  among  us  that  the  "show  me"  attitude 
is  firmly  entrenched  from  Congress  (as  evidenced  by  PL-92-156) 
down  through  the  Office  of  the  Secretary  of  Defense  (per 
DODD  5000.1  and  5000.3)  to  each  level  in  the  review  and 
approval  chain.  There  is  the  ever  present  danger  of  over- 
reaction to  changing  guidance.  Large  increases  in  program 
costs  due  to  a production  gap  during  conduct  of  I0T&-E  are  a 
major  concern  if  such  over  reaction  should  occur.  I ques- 
tioned many  people  at  each  level  of  the  T&E  community  from 
OSD  to  AFTEC  and  found  an  acute  awareness  of  the  potential 
dangers  in  an  overly  restrictive  test  environment. 

Still  there  remain  two  distinct  camps  5 one  convinced  that 
initial  operational  testing  must  come  before  any  production 
release,  the  second  equally  convinced  that  limited  production 
commitments  are  necessary  to  keep  the  contractors'  production 
and  engineering  staffs  usefully  occupied  while  awaiting 
completion  of  IOT&E  ar.d  the  production  decision. 
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The  first  camp  seems  to  have  won  the  latest  round,  as 
Congress  deleted  the  fourth  B-l  production  money  from  DOD's 
1975  supplemental  funding  bill.  This  fourth  aircraft  had 
been  requested  by  the  Air  Force  and  DOD  specifically  to  keep 
the  North  American  Rockwell  production  facility  occupied 
while  ICT&E  was  being  accomplished.  The  fact  that  a fourth 
aircraft  was  even  included  in  the  DOD  supplemental  funding 
request  shows  some  change  in  thinking  at  the  OSD  level. 

The  test  and  evaluation  evolution  we  are  witnessing 
today  is  not  unlike  the  wave  of  consumerism  which  has  been 
sweeping  our  country  for  the  past  several  years.  The 
motivations  are  the  same  - Congress,  representing  the  people 
of  our  country  is  tired  of  buying  systems  which  cost  too 

much  and  won't  perform  as  advertised.  Therefore,  in  addition 

. 

to  continuing  emphasis  on  program  cost  and  schedule  con- 
siderations, test  and  evaluation  must  take  on  new  importance. 
In  the  past  it  has  been  an  area  to  trade  off  for  dollar  or 
schedule  savings.  The  contrary  is  true  today,  with  additions 
to  the  planned  test  program  not  uncommon  (as  evidenced  by 
the  Maverick  program) . 

Recommendations  - Each  program  office  musti 
1,  Track  the  current  laws,  directives  and  regulations 
concerning  test  and  evaluation,  to  assess  and  prepare  for 
the  impact  on  their  program. 
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2.  Provide  detailed  planning  for  all  testing  phases 
(DT&E,  IOT&E  and  OT&E)  early  in  its  program  life  cycle. 

3.  Include  all  concerned  commands  and  agencies  in  all 
the  planning  phases. 

4.  Construct,  with  the  aid  of  AFTEC  and  other  concerned 
parties,  a test  plan  which  will  provide  convincing  proof 
that  its  system  will  do  the  mission  it  is  designed  to  do, 
and  do  it  in  its  operational  environment, 

5.  Assure  that  sufficient  numbers  of  its  system  are 
designed  into  its  full  scale  development  program  to  keep 
the  contractors’  production  and  engineering  personnel  in- 
tact until  IOT&E  is  complete  and  the  production  decision  is 
made. 

The  name  of  the  game  today  is  .thorough  testing  and  in- 
dependent evaluation.  T&E  is  the  final  selling  point  of 
each  program.  A good  test  program  will  not  sell  a bad 
system,  but  a poorly-designed  test  program  could  kill  an 
otherwise  good  system.  With  proper  attention  to  test  and 
evaluation  planning,  the  proving  of  a system's  worth  should 
progress  as  smoothly  as  the  acceptance  of  Aeroplane  No.  1. 
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APPENDIX  A 
GLOSSARY  OF  TERMS 


ACCEPTANCE  TESTS.  Those  tests  performed  to  demonstrate  that 
a specific  lot  of  articles  that  has  been  manufactured 
will  perform  within  the  tolerances  and  over  the  range  of 
operational  and  environmental  criteria  specified  for  the 
article. 

ACQUISITION  LIFE  CYCLE.  Normally,  it  consists  of  five 

phases;  (Conceptual,  Validation,  Full  Scale  Development, 
Production  and  Deployment)  with  decision  points  between 
each  of  the  first  four  phases  (Program,  Ratification, 
and  Production  Decisions). 

DEVELOPMENT  TEST  AND  EVALUATION  (DT&E) . Test  and  Evaluation 
which  focuses  on  the  technological  and  engineering  as- 
pects of  the  system,  subsystem,  or  equipment  items. 

EVALUATION.  The  review  and  analysis  of  qualitative  or 
quantitative  data  produced  during  current  or  previous 
testing,  or  operational  usage,  or  combination  thereof. 

IMPLEMENTING/DEVELOPING  COMMAND.  The  command  responsible 
for  the  acquisition  of  the  system,  subsystem,  or  item  of 
equipment. 

LOGISTICS  SUPP0RTAB1LITY.  How  well  the  composite  of 

support  considerations  necessary  to  achieve  the  effec- 
tive and  economical  support  of  a system  or  equipment 
for  its  life  cycle  meets  stated  quantitative  and 
qualitative  requirements. 


OPERATIONAL  TEST  AND  EVALUATION  (OT&E).  Test  and  Evaluation 
which  focuses  on  the  development  of  optimum  tactics, 
techniques,  and  concepts  for  systems  and  equipment, 
and  evaluation  of  reliability,  maintainability,  and 
operational  effectiveness  and  suitability  of  systems 
and  equipment  under  realistic  operational  conditions. 

(1)  INITIAL  OPERATIONAL  TEST  AND  EVALUATION  (IOT&E).  That 
OT&E  to  be  accomplished  prior  to  the  first  major  pro- 
duction decision  to  determine  operational  effectiveness, 
suitability  and  logistic  supportability, 

(2)  FOLLOW-ON  OPERATIONAL  TEST  AND  EVALUATION.  That  OT&E 
accomplished  subsequent  to  IOT&E  or  receipt  of  pro- 
duction items. 

OPERATING  COMMAND.  The  command  primarily  responsible  for 
the  operational  employment  of  a system,  subsystem,  or 
item  of  equipment.  This  term  generally  applies  to  those 
operational  commands  or  organizations  designated  by 
HQ  USAF  to  conduct  or  participate  in  operations  or 
operational  testing, 

OPERATIONAL  EFFECTIVENESS.  How  well  the  system  performs 
in  its  intended  environment  including  countermeasures. 

OPERATIONAL  SUITABILITY.  How  well  the  system  is  suited 
to  be  operated  and  maintained  by  military  personnel  in 
the  field. 


PRODUCTION  ACCEPTANCE  TEST  AND  EVALUATION  ( PAT&E ) . Test 
and  evaluation  of  production  items  to  demonstrate  that 
the  items  procured  fulfill  the  requirements  and  speci- 


fications of  the  procuring  contract  or  agreements 
PAT&E  continues  throughout  the  production  phase  of  the 
acquisition  process. 

PROGRAM  MANAGFR.  The  single  Air  Force  manager  during  any 
specific  phase  of  the  acquisition  life  cycle  (System 
Program  Director,  Program  Manager,  or  System  Manager/ 
Item  Manager). 


PROGRAM  OFFICE.  The  field  office  organized  by  the  Program 
Manager  to  assist  him  in  accomplishing  the  program  tasks, 
QUALIFICATION  TESTS.  Those  tests  that  verify  the  design 
and  manufacturing  process  and  thus  provide  a baseline 
for  subsequent  acceptance  tests.  Qualification  testing 
is  conducted  to  accomplish  two  separate  functions i 
(1)  Environmental  tolerance  of  design  (Preproduction 
Qualification  Tests).  A series  of  formal  contractual 
tests  are  conducted  to  insure  design  integrity  over  the 


specified  operational  and  environmental  range.  The  test 
should  be  conducted  on  prototype  or  preproduction 
hardware  fabricated  to  the  proposed  production  design 
specifications  and  drawings.  These  tests  are  a constraint 
to  production  release  on  programs  which  involve  volume 
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(2)  Manufacturing  process  verification  (Production 
Qualification  Tests).  A series  of  formal  contractual 
tests  are  conducted  to  insure  the  effectiveness  of  the 
manufacturing  process,  equipment,  and  procedures. 

These  tests  should  be  conducted  on  a sample  taken  at 
random  from  the  first  production  lot,  and  should  be 
repeated  if  the  process  is  changed  significantly  and 
when  a second  or  alternate  source  is  brought  on  line. 
SUPPORTING  COMMAND.  A command  that  provides  direct  support 
to  a system  or  test  program.  Normally  refers  to  AFLC, 
USAFSS,  and  ATC  in  their  role  as  logistics  support  and 
training  organizations. 

TEST.  Any  program  or  procedure  which  is  designed  to  obtain, 
verify,  or  provide  data  for  the  evaluation  of« 
research  and  development  (other  than  laboratory  ex- 
periments); progress  in  accomplishing  development  ob- 
jectives; or  performance  and  operational  capability  of 
systems,  subsystems,  components,  and  equipment  items. 

TEST  DIRECTCR.  A person  assigned  to  plan  and  direct  test 
and  evaluation  at  a test  site. 

TEST  PLAN.  A management  document  which  describes  how  and 
when  specified  test  objectives  will  be  met. 
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